Information processing device and information processing method

ABSTRACT

A processing device and method is provided. According to an illustrative embodiment, the device and method is implemented by detecting a face region of an image, setting at least one action region according to the position of the face region, comparing image data corresponding to the at least one action region to the detection information for purposes of determining whether or not a predetermined action has been performed, and generating a notification when it is determined that the predetermined action has been performed.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority from Japanese Patent ApplicationNo. JP 2010-200264 filed in the Japanese Patent Office on Sep. 7, 2010,the entire content of which is incorporated herein by reference.

BACKGROUND

The present disclosure relates to an information processing device andan information processing method.

In recent years, functions of devices such as electronic devices and soforth have become more sophisticated as well as increasing in number,and the instructions which the user has to give to the device to userthe device are becoming more varied and complex. As such, many devicesemploy GUI (Graphical User Interface) as the user interface, wherebyinstructions can be given to the device more easily than in a case ofusing a device employing CUI (Character User Interface).

However, for users who are not accustomed to using a pointing devicesuch as a mouse for example, a GUI is not necessarily an interface withgood operability. Accordingly, there is demand for a user interface bywhich users can give instructions to devices more intuitively. This hasled to user interfaces being developed to improve user operability.Japanese Unexamined Patent Application Publication No. 2005-31799 is oneexample of a technique for detecting shapes and actions of part of thebody of a user or objects operated by the user, and performingprocessing according to the detected results.

SUMMARY

An information processing device to which related art relating to a userinterface for improving operability (hereinafter referred to simply as“related art”) has been applied (hereinafter referred to as “informationprocessing device according to the related art”) detects gesture actionsof the user using the arms, palms, fingers, and so forth, in a space, byperforming image processing of imaged images (moving images, hereinafterreferred to as “imaged images”). In the event that gesture actions ofthe user have been detected, the information processing device accordingto the related art performs processing corresponding to the detectedgesture actions of the user. Accordingly, the user of a informationprocessing device according to the related art can provide instructionsto the information processing device according to the related art in amore intuitive manner than a case of using a GUI, and accordingly, useroperability might be improved using the related art.

However, with the related art, shapes and actions of at least a part ofthe body of the user on the space, such as arms, palms, fingers or thelike, or objects operated by the user, are detected to detect thegesture actions of the user, so image processing has to be performed onthe entire imaged image. Accordingly, the more type of gesture actionsthere are to detect, and the more complex the gesture actions are, thehigher the processing capability demanded of the information processingdevice according to the related art performing image processing is. Now,even though the processing capabilities of information processingdevices such as PCs (Personal Computer) and so forth have improved inrecent years along with higher functionality, diverting a great amountof calculating resources to the user interface might result ininsufficient calculating resources for realizing other functions such asfunctions for controlling communication, for example. Also, in the eventof a case where calculating resources for realizing other functions hasbecome insufficient, processing relating to this function is notperformed until the insufficiency of calculating resources is resolved,so with information processing device according to the related arts, anundesirable situation can occur in which the user is made to wait for anunreasonable amount of time, or the function is not realized.Accordingly, with the case of using the related art, user operability isnot improved without fail, since user convenience may be sacrificed inthe process.

It has been found desirable to provide a new and improved informationprocessing device and information processing method whereby useroperability can be improved.

In view of the above, the present device and method is provided.According to an illustrative embodiment, the device and method isimplemented by detecting a face region of an image, setting at least oneaction region according to the position of the face region, comparingimage data corresponding to the at least one action region to thedetection information for purposes of determining whether or not apredetermined action has been performed, and generating a notificationwhen it is determined that the predetermined action has been performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram illustrating an example of actionregions which an information processing device according to anembodiment of the present disclosure sets;

FIG. 2 is an explanatory diagram illustrating an example of actionregions which the information processing device according to anembodiment of the present disclosure sets;

FIG. 3 is an explanatory diagram illustrating an example of actionregion setting processing with the information processing deviceaccording to an embodiment of the present disclosure;

FIG. 4 is an explanatory diagram illustrating an example of notificationto a user due to notification processing according to an embodiment ofthe present disclosure being performed;

FIG. 5 is an explanatory diagram illustrating an example of change of animaged image due to notification processing according to an embodimentof the present disclosure being performed;

FIG. 6 is an explanatory diagram illustrating another example of changeof an imaged image due to notification processing according to anembodiment of the present disclosure being performed;

FIG. 7 is an explanatory diagram illustrating another example ofnotification to a user due to notification processing according to anembodiment of the present disclosure being performed;

FIG. 8 is an explanatory diagram illustrating another example ofnotification to a user due to notification processing according to anembodiment of the present disclosure being performed;

FIG. 9 is a flowchart illustrating an example of processing relating toan operability improvement approach with the information processingdevice according to an embodiment of the present disclosure;

FIG. 10 is a flowchart illustrating an example of action determinationprocessing with the information processing device according to anembodiment of the present disclosure;

FIG. 11 is an explanatory diagram for describing advantages of detectingthat a predetermined action has been performed using individualdetection information, by the information processing device according toan embodiment of the present disclosure;

FIG. 12 is an explanatory diagram for describing advantages of detectingthat a predetermined action has been performed using individualdetection information, by the information processing device according toan embodiment of the present disclosure;

FIG. 13 is a flowchart illustrating another example of processingrelating to the operability improvement approach with the informationprocessing device according to an embodiment of the present disclosure;

FIG. 14 is a block diagram illustrating an example of the configurationof an information processing device according to a first embodiment ofthe present disclosure;

FIG. 15 is an explanatory diagram illustrating an example of thehardware configuration of an information processing device according toan embodiment of the present disclosure;

FIG. 16 is a block diagram illustrating an example of the configurationof an information processing device according to a second embodiment ofthe present disclosure; and

FIG. 17 is a block diagram illustrating an example of the configurationof an information processing device according to a third embodiment ofthe present disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

Preferred embodiments of the present disclosure will be described withreference to the appended drawings. Note that in the presentSpecification and drawings, components having substantially samefunctional configurations will be denoted with the same referencenumeral, and redundant description will be omitted.

Also, description will be made below according to the following order.

1. Approaches According to Embodiment of Present Disclosure

-   -   1-1. First Processing for Operability Improvement Approach    -   1-2. Second Processing for Operability Improvement Approach

2. Information Processing Device According to Embodiments of PresentDisclosure

3. Program According to Embodiment of Present Disclosure

1. Approaches According to Embodiment of Present Disclosure

Before describing the configuration of an information processing deviceaccording to an embodiment of the present disclosure (hereinafter, maybe referred to as “information processing device 100”), operabilityimprovement approaches according to an embodiment of the presentdisclosure will be described. In the following, description will be madeby way of a primary example of a configuration wherein an imaged imagethat has been imaged at an external device of the information processingdevice 100 is input to the information processing device 100, and theinformation processing device 100 processes the imaged image. Note thatthe configuration of the information processing device 100 according toan embodiment of the present disclosure is not restricted to the above,and an arrangement may be made wherein, for example, the informationprocessing device 100 has imaging functions, and the imaged image thathas been imaged using the imaging functions is processed. Also, animaged image according to an embodiment of the present disclosure is amoving image obtained by imaging (group of still images imagedconsecutively on a time axis). Also, processing relating to operabilityimprovement approaches according to an embodiment of the presentdisclosure described below can be understood to be processing relatingto an information processing method according to an embodiment of thepresent disclosure.

Overview of Operability Improvement Approaches

As described above, in a case of detecting gesture actions of the user(hereinafter may be referred to simply as “actions”) and performingprocessing corresponding to the detected actions, the user can giveinstructions to the information processing device more intuitively thanwith a case of using a GUI. Accordingly, the information processingdevice 100 detects user actions by performing image processing of imagedimages, and performing processing corresponding to the detected actions,thereby improving user operations.

Now, in the case of performing image processing as to an entire imagedimage as with a information processing device according to the relatedart, high processing capabilities are a prerequisite to perform imageprocessing. Thus, diverting a great amount of calculating resources tothe user interface might as with information processing device accordingto the related arts may result in insufficient calculating resources forrealizing other functions. Also, in the event of a case wherecalculating resources for realizing other functions has becomeinsufficient, an undesirable situation can occur in which the user ismade to wait for an unreasonable amount of time, or the function is notrealized.

Accordingly, with the information processing device 100 a regionincluding the face of the user (hereinafter referred to as “faceregion”) is detected from the imaged image, and action region in theimaged image for detecting user actions are set nearby the face regionidentified by detection (hereinafter may be referred to as “identifiedface region”). The information processing device 100 detectspredetermined actions of the user in the set action regions, andperforms processing as to predetermined actions that have been detected.Now, predetermined user actions according to an embodiment of thepresent disclosure are user actions which are the object ofdetermination, in which determination is made at the informationprocessing device 100 regarding whether or not an action has beenperformed. The information processing device 100 performs the abovedetermination based on, for example, an imaged image, andlater-described reference detection information or individualdetermination information.

By setting action regions as described above, the information processingdevice 100 can detect user actions by performing image processing on aregion which is a part of the imaged image, rather than performing imageprocessing on the entire imaged image as with information processingdevice according to the related arts. Now, with the informationprocessing device 100, the calculation load relating to the userinterface can be markedly reduced as compared to information processingdevice according to the related arts, so there is no concern ofinsufficient calculation resources for realizing other functions as withinformation processing device according to the related arts. Also, theinformation processing device 100 identifies the face region from imagedimages by performing face recognition processing, which is relativelylight processing implemented in devices which do not have processingcapabilities very high, such as digital cameras of cellular phones forexample, and sets action regions nearby the face region, so thecalculation load at the time of setting action regions is not great,either.

Accordingly, the information processing device 100 can preventoccurrence of undesirable situations which can happen with informationprocessing device according to the related arts such as described above,so user convenience is not deteriorated as with information processingdevice according to the related arts. Accordingly, the informationprocessing device 100 can improve user operability.

Also, the information processing device 100 performs processingcorresponding to predetermined actions detected in the action regionssent nearby the face region, so erroneous detection of predeterminedactions due to actions performed by people or objects other than theuser corresponding to the face region for example, can be reduced ascompared to information processing device according to the related artswhich take the entire imaged image and the object of processing.Accordingly, the information processing device 100 can further improveuser operability as compared to information processing device accordingto the related arts.

Also, the information processing device 100 performs processingcorrelated to the meaning of the predetermined action, as processingcorresponding to the predetermined action detected in the action regionsset nearby the face region. For example, in the event of having detectedan action where the user reciprocally moves a hand sideways nearby theface (generally, an action meaning “good-bye” or “bye-bye”, hereinaftermay be referred to as “bye-bye action”) as the predetermined action, theinformation processing device 100 performs processing correlated to themeaning of the detected action, such as processing of turning offelectric power to a device to be controlled (information processingdevice or external device), muting audio (including music) output fromthe device, terminating communication at the device, or the like, forexample. Also, in the event of having detected an action where the userreciprocally moves a hand vertically in the action region as thepredetermined action, the information processing device 100 performsprocessing correlated to the meaning of the detected action, such asraising the volume of the audio output from a device to be controlled,or the like, for example.

As described above, by the information processing device 100 performingprocessing correlated to the meaning of detected action (processingtaking affordance into consideration), the chance that the user willsense inconsistency between the user's own actions and the processingwhich the information processing device 100 performs (or processingwhich the information processing device 100 causes an external device toperform) can be further reduced. Accordingly, the information processingdevice 100 can further improve user operability.

Further, in the event of determining that a predetermined action hasbeen performed (in the event that a predetermined action has beendetected), the information processing device 100 can further performnotification processing in which the user is notified that determinationhas been made at the information processing device 100 that apredetermined action has been performed. Also, in the event ofperforming notification processing, in a case of determination havingbeen made that a predetermined action has been continuously performedfor a predetermined period from determination having been made that thepredetermined action has been performed, the information processingdevice 100 performs processing corresponding to the predetermined actionthat has been determined.

Now, examples of notification processing according to an embodiment ofthe present disclosure include changing an image displayed on a displayscreen of a display unit (described later) which the device itself has,or changing the lit state of a lighting device provided to the displayunit, thereby visually performing notification to the user. That is tosay, notification processing according to an embodiment of the presentdisclosure can be understood as feedback processing to the user. Notethat the notification processing according to an embodiment of thepresent disclosure is not restricted to the above. For example, theinformation processing device 100 can perform processing for audiblyperforming notification to the user, such as outputting audio indicatingthat determination has been made that a predetermined action has beenperformed, from an audio output device which the device itself has(e.g., a later-described audio output unit) or an external audio outputdevice or the like, as notification processing. Also, The informationprocessing device 100 may transmit a processing execution command tonotify an external device that determination has been made at theinformation processing device 100 that a predetermined action has beenperformed, to an external device via a network (or directly), asnotification processing. By the above, the information processing device100 can cause an external device to perform notification to the user.

By the information processing device 100 performing notificationprocessing, the user can be made to comprehend that the action which theuser has performed has been recognized by the information processingdevice 100. Also, in the event that determination has been made that thepredetermined action has been performed continuously for a predeterminedperiod, the information processing device 100 performs processingcorresponding to the determined predetermined action, so the user cancancel execution of the processing corresponding to this action at theinformation processing device 100 by quitting his/her own action.Accordingly, in the event the user desires to cancel execution ofprocessing corresponding to the action at the information processingdevice 100 after the user has performed the predetermined action and theinformation processing device 100 has recognized this, or even in theevent that the information processing device 100 has happened toerroneously detect an action which the user has performed, for example,the user can control executing of processing at the informationprocessing device 100 corresponding to this action, by his/her ownaction.

Accordingly, by performing notification processing and furtherselectively performing processing corresponding to a predeterminedaction determined in the event that determination has been made that thepredetermined action has bee continuously performed for a predeterminedperiod, the information processing device 100 can prevent processingunintended by the user from being executed due to erroneous detection,and accordingly can further improve user convenience and operability.

The information processing device 100 improves user operability byperforming processing such as described above for example. Morespecifically, the information processing device 100 performs the Morespecifically, the information processing device 100 performs

1-1. first processing for operability improvement approach and

1-2. second processing for operability improvement approach for example,thereby improving user operability.

1-1. First Processing for Operability Improvement Approach

First, the first processing for the operability improvement approachaccording to an embodiment of the present disclosure will be described.The information processing device 100 performs the following processingof (1) through processing of (4), thereby improving user operability.

(1) Face Region Detection Processing

The information processing device 100 detects the face of the user(subject) from the imaged image, and identifies a face region. Now,identifying the face region can be performed by detecting feature pointsof, for example, the user's eyes, nose, mouth, structure, and so forth,and detecting facial brightness distribution and regions similar to astructural pattern. Note that the face region processing at theinformation processing device 100 according to an embodiment of thepresent disclosure is not restricted to the above. For example, theinformation processing device 100 can detect the difference between acurrent frame to be processed and one frame back to detect a movingbody, and detect the face region by identifying the head from the movingbody.

Also, data relating to identification of the face region at theinformation processing device 100, such as data of face brightnessdistribution and structure pattern and so forth, is stored in a storageunit (later described) which the information processing device 100 has.Note that the information processing device 100 can performcommunication with an external device such as a server via acommunication unit (later described) which the information processingdevice 100 has for example, and obtain data relating to identificationof a face region from the external device.

Also, as shown in FIG. 1 for example, the information processing device100 identifies a region greater than the face and including the detectedface, but the face region identifying method at the informationprocessing device 100 is not restricted to the above. For example, theinformation processing device 100 may identify the smallest region ofthe region including the detected face (e.g., a region closelycontaining the detected face such as shown in FIG. 3), as the faceregion.

The information processing device 100 identifies the face region asdescribed above, for example. Now, the imaged image which theinformation processing device 100 processes is a moving image, there canbe cases where the position in the imaged image of the face regionidentified in a certain frame changes in the next frame. Accordingly,the information processing device 100 performs face region detectionprocessing for each frame, for example. Note that the frequency ofperforming face region detection processing according to an embodimentof the present disclosure is not restricted to the above, and theinformation processing device 100 may perform face region detectionprocessing every predetermined frame, or every predetermined time, forexample.

(2) Action Region Setting Processing

Upon the face region being detected in the above processing of (1) (faceregion detection processing), the information processing device 100 setsaction regions nearby the face region, based on the face region.

FIGS. 1 and 2 are explanatory diagrams illustrating an example of theaction regions which the information processing device 100 according toan embodiment of the present disclosure sets. FA in FIGS. 1 and. 2denotes the face region of the user which the information processingdevice 100 has detected, and MA1 and MA2 in FIG. 1, and MA1 through MA4in FIG. 2, illustrate an example of the action region which theinformation processing device 100 has set. Hereinafter, the face regionmay be referred to as “face region FA”, and the action regions may becollectively referred to as “action region MA”. Now, the face region FAand action region MA shown in FIGS. 1 and 2 have been illustrated forsake of convenience of description, and in the event the imaged image isto be displayed on a display screen of an external device such as adisplay device 300 for example, or on a display screen of a display unit(later described) which the information processing device 100 has, theface region FA and action region MA do not have to be explicitly shown(this holds true throughout the following description). It goes withoutsaying that the information processing device 100 according to anembodiment of the present disclosure may display an imaged image inwhich the face region FA and action region MA are explicitly shown onthe display screen.

Now, while FIG. 1 illustrates an example where the informationprocessing device 100 has set circular action regions, the actionregions MA set by the information processing device 100 according to anembodiment of the present disclosure are not restricted to the exampleshown in FIG. 1. For example, the information processing device 100 mayset action regions of various shapes, such as rectangular action regionsas those shown in FIG. 2, elliptical action regions, and so forth. Also,while FIG. 1 shows a case where the information processing device 100has set action regions MA nearby the left side of the face region FA andnearby the right side of the face region FA, the action regions MA whichthe information processing device 100 according to an embodiment of thepresent disclosure sets are not restricted to the above-described. Forexample, the information processing device 100 may set an action regionMA to one of nearby the left side of the face region FA or the nearbythe right side of the face region FA. Also, as shown in FIG. 2 forexample, the information processing device 100 may further set one ormultiple action regions MA to nearby below the face region FA. Also, theinformation processing device 100 may set one or multiple action regionsMA to one of nearby below the face region FA or nearby above the faceregion FA, or to both nearby below the face region FA and nearby abovethe face region FA. It goes without saying that the action regions MAwhich the information processing device 100 sets near the face regionare not restricted to the above-described.

By setting action regions MA such as shown in FIGS. 1 and 2, theinformation processing device 100 does not detect an action performedoutside of an action region MA, such as indicated by A in FIG. 1 forexample, as a predetermined action. Accordingly, the informationprocessing device 100 can reduce the probability of erroneouslydetecting a predetermined action more than with a information processingdevice according to the related art which takes the entire imaged imageas the object of processing.

Also, by setting multiple action regions MA nearby the face region FA asshown in FIG. 2 for example, the information processing device 100 candetect predetermined actions in each of the multiple action regions MA,and perform processing for each action region MA corresponding to thedetected predetermined actions, for example.

Now, in the event of increasing the number of action regions MA to beset as shown in FIG. 2, for example, the calculation load is greaterthan the case shown in FIG. 1, but in comparison with a informationprocessing device according to the related art which performs imageprocessing on the entire imaged image, the calculation load is markedlyreduced. Accordingly, event in the event of increasing the number ofaction regions MA to be set as shown in FIG. 2 for example, there is noconcern of insufficient calculation resources as with informationprocessing device according to the related arts, so there is nodeterioration in user convenience.

Also, by performing processing corresponding to a predetermined actionfor each action region MA as described above, the information processingdevice 100 can perform processing different for each detected actionregion MA even in the event that the user has performed the samepredetermined action, for example. That is to say, the informationprocessing device 100 can appropriate multiple processing for onepredetermined action. It goes without saying that the processing whichthe information processing device 100 performs in the event that onepredetermined action is detected at each of the multiple action regionsMA may be the same.

Further, by performing processing corresponding to predeterminedoperations for each action region MA as described above, the informationprocessing device 100 can perform processing correlated with the meaningof the detected action (processing taking affordance intoconsideration), coupled with the position where the predetermined actionwas detected (more specifically, in which region it was detected).Accordingly, the information processing device 100 can further improveuser operability.

Specific Example of Action Region Setting Processing

Now, an example of action region setting processing with the informationprocessing device 100 will be described in further detail. FIG. 3 is anexplanatory diagram for describing an example of action region settingprocessing at the information processing device 100 according to anembodiment of the present disclosure. Now, FIG. 3 illustrates an exampleof processing in a case of setting action regions MA to nearby the leftside and nearby the right side of the face region FA, as shown in FIG.1.

The information processing device 100 sets action regions MA based onthe width (length in horizontal direction) w and height (length invertical direction) h of the identified face region FA. Now, FIG. 3illustrates an example where the information processing device 100 hasset rectangular action regions MA1 and MA2 with a width of 2w and aheight of 2h. Note that the information processing device 100 can alsoset the action regions MA so as to be ellipses where 2w and 2h are theminor axis and major axis, for example (a circle when w=h).

Also, the information processing device 100 sets the action regions MAso as to come into contact with the identified face region FA as shownin FIG. 3 for example, but the setting method of the action regions MAby the information processing device 100 is not restricted to theabove-described. For example, the information processing device 100 mayset the action regions MA at positions distanced from the identifiedface region FA by a predetermined number of pixels. That is to say, to“set an action region nearby a face region” according to an embodimentof the present disclosure includes setting an action region MA incontact with a face region FA and setting an action region MA at aposition in close proximity to a face region FA.

As described above, for example, the information processing device 100sets the action regions MA based on the width w and height h of theidentified face region FA.

Note that the setting processing of action regions MA with theinformation processing device 100 according to an embodiment of thepresent disclosure is not restricted to the above-described. Forexample, the information processing device 100 may perform facerecognition processing on the image within the face region FA in theimaged image to estimate user attributes such as age and gender and thelike of the user with regard to this face region FA, and adjust theregion size and position of the action regions MA, based on theestimation results. By adjusting the region size and position of theaction regions MA based on the estimation results as described above,action regions MA can be set corresponding to the age and gender of theuser, such as setting the position of the action regions MA shown inFIG. 2 so as to be lower for aged users, for example. Note that whilethe information processing device 100 estimates user attributes byextracting attribute information such as structure, wrinkles, saggingskin, and so forth for example, from the face region FA and analyzingthis, the method for estimating user attributes according to anembodiment of the present disclosure is not restricted to theabove-described. For example, the information processing device 100 mayestimate user attributes using optional techniques capable of estimatinguser attributes such as age and gender and the like of the usercorresponding to the face region FA.

Also, the information processing device 100 can extract face informationindicating features of the face of the user by performing facerecognition processing to the image in the face region FA, and adjustthe action regions MA by identifying the user corresponding to the faceregion FA based on the extracted face information. More specifically, inthe event that the user has been identified based on the faceinformation, the information processing device 100 sets action regionsMA for the identified user. An example of information relating tosetting of action regions MA for the user include a database where faceinformation and information relating to adjustment of the action regionsMA is recorded. Also, information relating to setting of the actionregions MA of the user may be included in later-described individualdetection information. The information processing device 100 can obtainface information indicating the features of the face of the user thathas been extracted, but extracting facial features of the user (e.g.,structure information, information indicating the positional relation ofeyes, nose, and mouth, and so forth) by performing image processing ofthe face region FA, for example. Also, the information processing device100 obtains information relating to adjustment of the action regions MA,such as adjustment amount of region size of the action regions MA,position adjustment amount of action regions MA, and so forth, based onuser operations, for example. Note that the information relating toadjustment of action regions MA may include information indicating thenumber of action regions MA set. By setting action regions MA for usersas described above, the action regions MA which the users themselves canadjust can be set, so the information processing device 100 can furtherincrease the freedom of the user, and improve convenience.

By performing processing such as described above for example, theinformation processing device 100 sets action regions MA nearby the faceregion FA based on the face region FA. Now, as described above, theimaged image which the information processing device 100 processes is amoving image, so the position of the face region in the imaged imageidentified in a certain frame may change in the next frame. Theinformation processing device 100 performs setting of the action regionsMA based on the face region FA each time a face region FA is identifiedby the processing of (1) above, but the action region setting processingby the information processing device 100 according to an embodiment ofthe present disclosure is not restricted to the above-described. Forexample, the information processing device 100 may detect the amount ofmovement of the face region FA in the current frame as to the faceregion FA in the previous frame, and in the event that the amount ofmovement is equal to or smaller than a predetermined threshold (orsmaller than the threshold), sets the action region MA set in theprevious frame so as to be moved to corresponding to the amount ofmovement. It goes without saying that the face region FA in the previousframe is not restricted to one frame back from the current frame (forexample, in the event that the processing of (1) is to be performedevery predetermined frame, the frame before the predetermined frame isthe previous frame).

(3) Action Determination Processing

Upon action regions MA being set in the processing of (2) above (actionregion setting processing), the information processing device 100determines that a predetermined action has been performed by the user inan action region MA, based on the imaged image, and the referencedetection information or individual detection information.

Now, reference detection information according to an embodiment of thepresent disclosure is data serving as a reference of detection ofpredetermined actions of the user (object of detection). Here, theinformation processing device 100 stores the reference detectioninformation beforehand by this being recorded in the informationprocessing device 100 at the time of manufacturing by the manufacturerof the information processing device 100, but the information processingdevice 100 is not restricted to a configuration of storing referencedetection information beforehand. For example, the informationprocessing device 100 may obtain reference detection informationgenerated by an external device such as a server or the like, from theexternal device. Examples of reference detection information includedata and databases where predetermined actions and periodic componentinformation for detecting periodic actions are correlated. Also,examples of periodic component information according to an embodiment ofthe present disclosure include information of frequency of change inbrightness in the imaged image, but periodic component informationaccording to an embodiment of the present disclosure is not restrictedto the above. Note that reference detection information according to anembodiment of the present disclosure is not restricted to the above, andmay be data and databases where predetermined actions, and informationfor distinguishing gestures of the user which are not periodic actions(e.g., data for determining that the shape of the hand of the user is apredetermined shape), are correlated. Hereinafter, description will bemade with reference to an example where reference detection informationis a data base in which predetermined actions and periodic componentinformation are correlated for each predetermined action and recorded.Note that while examples of the database according to an embodiment ofthe present disclosure include relational databases, the databaseaccording to an embodiment of the present disclosure is not restrictedto the above.

Also, individual detection information according to an embodiment of thepresent disclosure is data to serve as a reference for detection ofpredetermined actions which a particular user performs. The individualdetection information is generated and updated by a learning unit (laterdescribed), for example. Note that the individual detection informationaccording to an embodiment of the present disclosure may be generated byan external device such as a server, with the information processingdevice 100 obtaining the individual detection information from theexternal device. Now, examples of individual detection informationinclude predetermined actions, data and databases where periodiccomponent information for detecting periodic actions of the user, andface information indicating facial features of the user (information foridentifying the user), are correlated. Note that the individualdetection information according to an embodiment of the presentdisclosure is not restricted to the above-described, and may be data anddatabases where predetermined actions, information for distinguishinggestures of the user which are not periodic actions, and facialinformation, are correlated. Hereinafter, description will be made withreference to an example where the individual detection information is adatabase where predetermined actions, periodic component information,and facial information, are correlated for each predetermined action,and recorded.

(3-1) First Example of Action Determination Processing

Processing based on reference detection information will be described asa first example of action determination processing according to anembodiment of the present disclosure. The information processing device100 detects a periodic component in an action region MA set based on theimaged image. The information processing device 100 then determineswhether or not a predetermined action has been performed, based on thedetection results at the action region MA and periodic componentinformation of the reference detection information. Now, the informationprocessing device 100 determines whether or not a predetermined actionhas been performed by analyzing properties of a specific point (specificpixel) within the action region MA, using the Fischer's method or thelike for example, but the determination method according to anembodiment of the present disclosure is not restricted to the above.Also, while the information processing device 100 detects the periodiccomponents in the action region MA using imaged images everypredetermined time or imaged images in increments of a predeterminednumber of frames, for example, the detection method for periodiccomponents in the action region MA according to an embodiment of thepresent disclosure is not restricted to the above-described.

(3-2) Second Example of Action Determination Processing

Next, processing selectively using reference detection information orindividual detection information will be described as a second exampleof action determination processing according to an embodiment of thepresent disclosure. The information processing device 100 extracts faceinformation from the face region FA identified by the processing of (1)above. Also, the information processing device 100 determines whether ornot there is face information corresponding to the face informationextracted from the individual detection information, based on theextracted face information and individual detection information. Now, inthe event that the face information according to an embodiment of thepresent disclosure is a group of multiple information, such asinformation of the structure and information indicating the positionalrelation of the eyes, nose, and mouth, for example, the informationprocessing device 100 determines that there is face informationcorresponding to the extracted face information in the event that thereis face information matching the extracted face information in theindividual detection information. Note that the determination methodregarding whether or not there is face information corresponding to theextracted face information in the individual detection informationperformed at the information processing device 100 according to anembodiment of the present disclosure, is not restricted to theabove-described. For example, the information processing device 100 maydetermine that there is face information corresponding to the extractedface information in the event that there is face information matchingface information regarding which a predetermined number or greater ofthe group of multiple information has been extracted.

In the event of having determined that there is individual detectioninformation corresponding to the extracted face information, theinformation processing device 100 determines whether or not apredetermined action correlated with the periodic component informationhas been made, based on the detection results of the periodic componentin the action region MA, and the periodic component information includedin the individual detection information.

Also, in the event that determination is not made that there isindividual detection information corresponding to the extracted faceinformation, the information processing device 100 determines whether ornot a predetermined action has been performed, based on the detectionresults and periodic component information of the reference detectioninformation, in the same way as with the first example of actiondetermination processing described above.

As described above for example, the information processing device 100determines that a predetermined action has been performed by the user inan action region MA, based on the imaged image and reference detectioninformation or individual detection information, as described above forexample. Note that the action determination processing at theinformation processing device 100 according to an embodiment of thepresent disclosure is not restricted to the above. For example, in theevent of performing determination using information for distinguishinggestures of the user which are not periodic components, for example, theinformation processing device 100 can determine that a predeterminedaction has been performed by the user in an action region MA, based onwhether or not the shape of the hand of the user in the action region MAis a predetermined shape stipulated in information for distinguishingthe gestures in the reference detection information or individualdetection information.

Also, in the event that determination is made that a predeterminedaction has been performed, the information processing device 100 canalso update the individual detection information (so-called learning).More specifically, in the event that determination is made that apredetermined action has been performed, the information processingdevice 100 takes periodic component information detected based on theimaged image, for example, as periodic component information in theindividual detection information corresponding to the determinedpredetermined action and the face information extracted from the imagedimage. Also, in the event that determination is made that apredetermined action has been performed, the information processingdevice 100 may take information representing the shape of the hand ofthe user that has been detected (i.e., information of the position ofjoints, etc.), for example, as information for distinguishing thegestures in the individual detection information. By updating theindividual detection information as described above, the informationprocessing device 100 can further improve the determination precision ofpredetermined actions of a particular user of which predeterminedactions are determined from individual detection information, and thusfurther reduce the possibility or erroneous determination.

Now, the information processing device 100 performs the above-describedprocessing relating to updating of individual detection information eachtime determination is made that a predetermined action has beenperformed (processing relating to learning), for example, but processingat the information processing device 100 is not restricted to the above.For example, the information processing device 100 can selectivelyperform processing relating to updating of the individual detectioninformation (processing relating to learning), based on user operations.

Also, in the event of determining whether or not a predetermined actionhas been performed using periodic component information, the informationprocessing device 100 determines whether or not a predetermined actionhas been performed each time a periodic component is detected in anaction region MA for example, but the action determination processingaccording to an embodiment of the present disclosure is not restrictedto the above. For example, the information processing device 100 maydetect whether or not there is a moving body in an action region MA setbased on the imaged image every predetermined time or imaged images inincrements of a predetermined number of frames, and selectivelydetermine that a predetermined action has been performed in the eventthat a moving body is detected. Now, in the event that no moving body isdetected in an action region MA when determining whether or not apredetermined action has been performed using periodic componentinformation, the probability that the user has not performed an actionin action region MA is high. Accordingly, by selectively detectingwhether or not a predetermined action has been performed in the eventthat a moving body is detected as described above, the informationprocessing device 100 can reduce calculation load on the informationprocessing device 100 while preventing missed detection of predeterminedactions of the user and erroneous detection.

(4) Execution Processing

Upon determination being made that a predetermined action has beenperformed in the processing of (3) above (action determinationprocessing), the information processing device 100 performs processingcorresponding to the determined predetermined action.

More specifically, the information processing device 100 storesprocessing information in which the predetermined action and processingto be executed have been recorded in a correlated manner for example,and performs processing correlated with the determined predeterminedaction in the processing of (3) above, based on the processing resultsof (3) above and the processing information. Note that the processinginformation according to an embodiment of the present disclosure is notrestricted to the above-described. For example, the informationprocessing device 100 may store processing information in whichinformation indicating an action region MA (e.g., region No. or thelike), predetermined action, and processing to be executed, recorded ina correlated manner. In the case described above, the informationprocessing device 100 can perform processing corresponding to apredetermined action for each action region MA regarding whichdetermination has been made that a predetermined action has beenperformed. Also, the processing information according to an embodimentof the present disclosure may be, for example, data for eachpredetermined action, or may be a database in which is recordedinformation relating to multiple predetermined actions.

Also, while an example of processing corresponding to a predeterminedaction which the information processing device 100 performs includesprocessing relating to functions which the device itself has, theprocessing corresponding to the predetermined actions which theinformation processing device 100 according to an embodiment of thepresent disclosure performs is not restricted to the above. For example,as processing corresponding to a predetermined action, the informationprocessing device 100 may transmit a processing execution command forcausing an external device to perform processing corresponding to apredetermined action, via a network (or directly) to the externaldevice. Thus, the information processing device 100 can cause theexternal device to perform processing corresponding to predeterminedactions of the user.

The information processing device 100 performs processing of (1) (faceregion detection processing) through processing of (4) (executionprocessing) described above for example, as first processing relating tothe operability improvement approach, thereby determining that the userhas performed a predetermined action, and perform processingcorresponding to the determined predetermined action.

Now, the information processing device 100 sets action regions MA nearbythe face region FA identified based on the imaged image, detects(determines) predetermined actions of the user in the set action regionsMA, and performs processing corresponding to the detected predeterminedactions. By setting the action regions MA as described above, theinformation processing device 100 can detect predetermined actions ofthe user by performing image processing as to a partial region of theimaged image, rather than performing image processing on the entireimaged image as with information processing device according to therelated arts. Accordingly, the information processing device 100 canreduce the calculation load relating to detection of predeterminedactions by the user as compared to information processing deviceaccording to the related arts, so with the information processing device100, occurrence of undesirable situations which can happen withinformation processing device according to the related arts such asdescribed above, can be prevented. Accordingly, the informationprocessing device 100 can prevent deterioration in user conveniencewhich can occur in a case of using the related art, and also, useroperability can be improved.

1-2. Second Processing for Operability Improvement Approach

The information processing device 100 can improve user operability byperforming the first processing for the operability improvement approachdescribed above. However, the processing for the operability improvementapproach is not restricted to the above-described. For example, asdescribed above, the information processing device 100 can performprocessing corresponding to a predetermined action determined in thecase of having performed notification processing whereupon determinationis made that the predetermined action has been performed continuouslyfor a predetermined period. Accordingly, next, processing in a case ofperforming notification processing will be described as the secondprocessing for the operability improvement approach according to anembodiment of the present disclosure. The information processing device100 performs the following processing of (I) through processing of (V)for example, thereby preventing processing unintended by the user due toerroneous detection from being performed, and further improving userconvenience and operability.

(I) Face Region Detection Processing

In the same way as with the processing of (1) above (face regiondetection processing), the information processing device 100 detects theface of the user from in the imaged image, and determines the faceregion FA.

(II) Action Region Setting Processing

Upon the face region being detected in the processing of (1) above (faceregion detection processing), the information processing device 100 setsaction regions MA nearby the face region FA, based on the face regionFA, in the same way as with the processing of (2) above (action regionsetting processing).

(III) Action Determination Processing

Upon setting of the action regions MA in the processing of (II) above(face region detection processing), the information processing device100 determines that a predetermined action by the user has beenperformed in an action region MA, based on the imaged image andreference detection information or individual detection information, inthe same way as with the processing of (3) above (action determinationprocessing).

Also, the information processing device 100 determines whether or not apredetermined action has been continuously performed for a predeterminedperiod from determination having been made that the predetermined actionhas been performed. Now, an example of the predetermined period from thedetermination having been made includes the time for performing theprocessing of (IV) described later, but the predetermined period fromthe determination having been made according to an embodiment of thepresent disclosure is not restricted to be above-described. For example,the predetermined period from determination having been made accordingto an embodiment of the present disclosure can be arbitrarily set by themanufacturer of the information processing device 100 or the like, suchas for example, time of the time for performing the processing of (IV)described later plus leeway time.

(IV) Notification Processing

Upon determination being made that a predetermined action has beenperformed in an action region MA in the processing of (III) above(action determination processing), the information processing device 100performs notification processing.

Specific Example of Notification to User

Now, a specific example of notification to the user by notificationprocessing according to an embodiment of the present disclosure havingbeen performed will be described. In the following, description will bemade with reference to an example of a case of performing notificationprocessing for visually notifying the user.

FIG. 4 is an explanatory diagram illustrating an example of notificationto the user by notification processing according to an embodiment of thepresent disclosure being performed. Here, FIG. 4 illustrates an examplewhere the imaged image is displayed on a display screen (for example, adisplay screen of an external device such as a display device 300 or adisplay unit (later described) which the information processing device100 has, hereinafter, the same), and visual notification is made thatdetermination has been made that a predetermined action has beenperformed, by changing the displayed imaged image, as shown in B in FIG.4. That is to say, FIG. 4 illustrates an example of a case of visuallymaking notification that the information processing device 100 hasdetermined that the predetermined action has been performed, by changingthe image displayed on the display screen.

FIG. 5 is an explanatory diagram illustrating an example of change of animaged image by the notification processing according to an embodimentof the present disclosure being performed. FIG. 5 illustrates an exampleof changing the imaged image by the information processing device 100superimposing a notification object on the action region MA1 (the actionregion regarding which determination has been made that the user hasperformed a predetermined action) of the imaged image. Here, in FIG. 5,an example is illustrated where the information processing device 100superimposes multiple shining stars on the imaged image, but thenotification object according to an embodiment of the present disclosureis not restricted to the above-described. The user can visuallycomprehend that the information processing device 100 has determinedthat the user has performed a predetermined action in the action regionMA, by seeing the change in the imaged image displayed on the displayscreen.

Note that the change in the imaged image due to notification processingaccording to an embodiment of the present disclosure being performed isnot restricted to the notification object shown in FIG. 5 beingsuperimposed. FIG. 6 is an explanatory diagram illustrating anotherexample of change of the imaged image due to the notification processingaccording to an embodiment of the present disclosure being performed. Asshown in FIG. 6, the imaged image may be changed by highlight processingusing a round mask or the like being performed toward the face region FAcorresponding to the action region regarding which determination hasbeen made that the user has performed the predetermined action in theimaged image. In this case as well, the user can visually comprehendthat the information processing device 100 has determined that the userhas performed a predetermined action in the action region MA, by seeingthe change in the imaged image displayed on the display screen.

The information processing device 100 visually notifies thatdetermination has been made that a predetermined action has been made,by changing the imaged image displayed on the display screen asdescribed above, for example. Accordingly, the user can cancel executionof the processing corresponding to the action at the informationprocessing device 100 by stopping his/her own action, so the informationprocessing device 100 can prevent processing which the user does notintend due to erroneous detection from being executed, and userconvenience and operability can be further improved.

Note that notification to the user by notification processing accordingto an embodiment of the present disclosure being performed is notrestricted to changing the imaged image displayed in the display screenas shown in FIGS. 4 through 6. FIGS. 7 and 8 are explanatory diagramsillustrating other examples of notification user by notificationprocessing according to an embodiment of the present disclosure beingperformed.

As indicated by C in FIG. 7, the information processing device 100 canchange the image by superimposing an icon on an image displayed on adisplay screen, such as an image illustrating contents for example. Notethat the method for changing an image displayed on the display screenaccording to an embodiment of the present disclosure is not restrictedto superimposing the icon shown in FIG. 7. For example, the informationprocessing device 100 may superimpose various objects, such as anindicator displaying time in stages from detecting the predeterminedaction by the user till starting the processing corresponding to thepredetermined action, or the like, on the image displayed on the displayscreen. Also, the information processing device 100 may display theobject such as an icon or indicator outside of the display region wherethe image indicating the content on the display screen, rather thansuperimposing the object on the image displayed on the display screen.In the above case as well, the user can visually comprehend that theinformation processing device 100 has determined that the user hasperformed a predetermined action in the action region MA, by seeing thechange in the imaged image displayed on the display screen.

Also, as indicated by D in FIG. 8, visual notification can be made thatdetermination has been made that a predetermined action has beenperformed, by a lighting device such as an LED (Light Emitting Diode)provided on the outer portion of the display screen being selectivelylit (or turned off). Also, the information processing device 100 canmake visual notification that determination has been made that apredetermined action has been performed, by changing the color of lightemitted from the lighting device. As described above, the user canvisually comprehend that the information processing device 100 hasdetermined that the user has performed a predetermined action in theaction region MA, by the information processing device 100 changing thelit state of the lighting device which an external device such as adisplay device 300 or a display unit (later described) which theinformation processing device 100 has.

As described above for example, the information processing device 100can visually notify that determination has been made that apredetermined action has been performed, by changing the image displayedon the display screen, or changing the lit state of the lighting device.Note that while changing of the image displayed on the display screen(including the imaged image) and changing the lit state of the lightingdevice have been illustrated individually in the above description, butit goes without saying that the information processing device 100according to an embodiment of the present disclosure can make visualnotification by combining these.

The information processing device 100 visually notifies the user (ornotifies to an external device) as described above for example, that theinformation processing device 100 has determined that a predeterminedaction has been performed by the user in an action region MA, byperforming notification processing. Also, in the event that apredetermined action is performed continuously from a predeterminedperiod from determination having been made that the predetermined actionhas been made, the information processing device 100 performs processingcorresponding to the predetermined action in the later-describedprocessing of (V) (execution processing). Accordingly, the user cancancel execution of the processing corresponding to the action at theinformation processing device 100 by stopping his/her own action, so theinformation processing device 100 can prevent processing which the userdoes not intend due to erroneous detection from being executed, and userconvenience and operability can be further improved. Note that while anexample of the information processing device 100 visually notifying theuser by performing notification processing has been illustrated above,it goes without saying that the same advantages can be obtained even incases of performing notification by other methods, such as a case of theinformation processing device 100 audibly performing notification byperforming notification processing.

(V) Execution Processing

Upon determination being made in the processing of (III) above (actiondetermination processing) that a predetermined action has been performedcontinuously for a predetermined period since determination was madethat he predetermined action was performed, the information processingdevice 100 performs processing corresponding to the determinedpredetermined action. Now, the information processing device 100performs processing corresponding to the determined predeterminedaction, in the same way as with the processing of (4) above (executionprocessing).

As the second processing relating to the operability improvementapproach, the information processing device 100 performs the processingof (I) (face region detection processing) through (V) (executionprocessing) described above, thereby determining that a predeterminedaction has been performed by the user, and performing processingcorresponding to the predetermined action.

Now, the second processing relating to the operability improvementapproach has basically the same processing as the first processingrelating to the first operability improvement approach. Accordingly, byperforming the second processing relating to the operability improvementapproach, the information processing device 100 prevent deterioration inuser convenience which can occur in the event of using the related art,and also improvement in user operability can be improved, in the sameway as performing the processing relating to the first operabilityimprovement approach.

Also with the second processing relating to the operability improvementapproach, the information processing device 100 notifies the user (ornotifies to an external device) that the information processing device100 has determined that a predetermined action has been performed by theuser in an action region MA, by the processing of (IV). Also, in theevent that a predetermined action is performed continuously from apredetermined period from determination having been made that thepredetermined action has been made, the information processing device100 performs processing corresponding to the predetermined action in theprocessing of (V) (execution processing). Accordingly, the user cancancel execution of the processing corresponding to the action at theinformation processing device 100 by stopping his/her own action, so theuser can control execution of processing corresponding to thepredetermined action at the information processing device 100 by his/herown actions. Accordingly, the information processing device 100 canprevent processing which the user does not intend due to erroneousdetection from being executed, and user convenience and operability canbe further improved.

Specific Examples of Processing Relating to Operability ImprovementApproach

Next, an example of processing for realizing the operability improvementapproach according to an embodiment of the present disclosure describedabove will be described.

1-1-1. Specific Example of First Processing Relating to OperabilityImprovement Approach

FIG. 9 is a flowchart illustrating an example of processing relating tothe operability improvement approach at an information processing device100 according to an embodiment of the present disclosure, andillustrates an example of processing corresponding to the firstprocessing relating to the operability improvement approach. In thefollowing, description will be made with reference to an example ofprocessing in a case of the information processing device 100determining a predetermined action made by a user, based on periodiccomponents in an action region MA set as to the imaged image.

The information processing device 100 determines whether or not toperform action detection of the user (S100). Now, in a case there havebeen provided functions for selectively switching between a firstprocessing mode where the information processing device 100 performsaction detection and a second processing mode where action detection isnot performed, for example, based on user operations and the like, theprocessing in step S100 is equivalent to determination of whether or notthe processing mode is the first processing mode. That is to say, in theevent that the information processing device 100 does not have afunction for switching between the first processing mode and the secondprocessing mode, the information processing device 100 does not have toperform the processing of step S100.

In the event that determination is not made to perform action detectionof the user in step S100, the information processing device 100 ends theprocessing relating to the operability improvement approach. Note thatthe processing relating to the operability improvement approach in FIG.9 is not processing of a type which is not performed again once it isended, rather, the information processing device 100 canperiodically/non-periodically repeat the processing relating to thefirst operability improvement approach shown in FIG. 9 even though theprocessing ends for the time being.

Also, in the event that determination is made to perform actiondetection of the user in step S100, the information processing device100 performs face detection based on the input imaged image, andidentifies a face region FA (S102). Now, the processing in step S102corresponds to the processing of (1) above (face region detectionprocessing). Also, the information processing device 100 identifies theface region FA by detecting feature points such as, for example, theuser's eyes, nose, mouth, structure, and so forth, and detecting regionssimilar to the face brightness distribution and structure pattern;however, the method for identifying the face region FA is not restrictedto the above-described.

Also, in the event that multiple face regions FA are identified from theimaged image, such as cases where there are multiple users in the imagedimage, the information processing device 100 performs the processing ofthe later-described steps S104 through S112, for each of the identifiedmultiple face regions FA.

Upon performing the processing of step S102, the information processingdevice 100 determines whether or not a face region FA has beenidentified (S104). In the event that determination is not made that aface region FA has been identified in step S104, the informationprocessing device 100 repeats the processing from step S100.

Also, in the event that determination is made that a face region FA hasbeen identified in step S104, the information processing device 100 setsaction regions MA based on the face region FA (S106). Now, theprocessing in step S106 corresponds to the processing of (2) above(action region setting processing). Also, the information processingdevice 100 sets the action regions MA nearby the face region FA as shownin FIGS. 1 and 2 for example, but the action regions MA set by theinformation processing device 100 are not restricted to the examplesshown in FIGS. 1 and 2.

Upon action regions MA being set (or re-set) in step S106, theinformation processing device 100 detects predetermined actions in theset action regions MA (S108). The information processing device 100 thendetermines whether or not a predetermined action has been detectedwithin the set action regions MA (S110). Here, the processing of stepsS108 and S110 corresponds to the processing of (3) above (actiondetermination processing).

More specifically, in step S108 the information processing device 100detects whether or not a predetermined action has been performed, basedon detection results of periodic components in the action regions MA setbased on the imaged image, and periodic component information ofreference detection information, for example (equivalent to firstexample of action determination processing described above). Now, thereference detection information is stored in a storage unit (describedlater) which the information processing device 100 has, for example,with the information processing device 100 reading out the referencedetection information from this storage unit to perform theabove-described detection, but the processing at the informationprocessing device 100 according to an embodiment of the presentdisclosure is not restricted to the above-described. For example, theinformation processing device 100 may perform communication with anexternal device such as a server storing reference detectioninformation, and obtain reference detection information from theexternal devices, so as to perform the detection using the obtainedreference detection information.

Note that the processing of step S108 at the information processingdevice 100 according to an embodiment of the present disclosure is notrestricted to processing using reference detection information. Forexample, the information processing device 100 may perform processingusing reference detection information or individual detectioninformation as the processing of step S108, as illustrated in the secondexample of the action determination processing described above.

FIG. 10 is a flowchart illustrating an example of action determinationprocessing at the information processing device 100 according to anembodiment of the present disclosure. Here, FIG. 10 illustrates anexample of processing equivalent to the second example of the actiondetermination processing described above.

The information processing device 100 extracts face information from theface region FA (S200). The information processing device 100 thendetermines whether or not there is face information in the individualdetection information that corresponds to the extracted face information(S202).

Now, the individual detection information is stored in a storage unit(described later) which the information processing device 100 has, forexample, with the information processing device 100 reading out theindividual detection information from this storage unit and performingthe above-described determination processing, but the processing at theinformation processing device 100 according to an embodiment of thepresent disclosure is not restricted to the above-described. Forexample, the information processing device 100 may perform communicationwith an external device such as a server storing individual detectioninformation, and obtain individual detection information from theexternal devices, so as to perform the detection using the obtainedindividual detection information. Also, the information processingdevice 100 may transmit the face information extracted in step S200 toan external device storing the individual detection information so as tocause the external device to perform actual determination regardingwhether or not there is face information in the individual detectioninformation corresponding to the extracted face information, and obtainthe determination results from the external device. In the above case aswell, the information processing device 100 can perform the processingof step S202 based upon the determination results obtained from theexternal device.

In the event that determination is made that there is face informationin the individual detection information corresponding to the extractedface information in step S202, the information processing device 100detects that a predetermined action has been performed, based on thedetection results of periodic components in the action region MA setbased on the imaged image, and periodic component information of theindividual detection information in which is included the faceinformation corresponding to the extracted face information.

Also, in the event that determination is not made in step S202 thatthere is face information in the individual detection informationcorresponding to the extracted face information, the informationprocessing device 100 detects that a predetermined action has beenperformed, based on the detection results of periodic components in theaction region MA set based on the imaged image, and periodic componentinformation of the reference detection information.

By performing the processing shown in FIG. 10 for example, theinformation processing device 100 can detect that a predetermined actionhas been performed using the reference detection information orindividual detection information.

Also, by the information processing device 100 detecting that apredetermined action has been performed using the individual detectioninformation, the information processing device 100 can prevent erroneousdetection or excessive detection of predetermined actions of userscorresponding to the face information included in the individualdetection information.

FIGS. 11 and 12 are explanatory diagrams for describing advantages ofthe information processing device 100 according to an embodiment of thepresent disclosure detecting that a predetermined action has beenperformed using individual detection information.

In the event that there are a user A and user B in an imaged image suchas shown in FIG. 11, part or all of the action regions MA of each usermay overlap action regions MA of the other user. In this case, in theevent that one user (the user B in FIG. 11) has performed a bye-byeaction in an overlapping action regions MA, which user has performedthis action may not be able to be detected unequivocally. Now, in theevent of performing determination using the individual detectioninformation, the information processing device 100 can unequivocallydetermine whether the action which has been performed is an action ofthe user A or an action of the user B. Accordingly, even in cases thatthere are the user A and user B in an imaged image such as shown in FIG.11, the information processing device 100 can prevent erroneousdetection or excessive detection of predetermined actions of userscorresponding to the face information included in the individualdetection information, by detecting that a predetermined action has beenperformed using the individual detection information.

Also, by performing determination using individual detectioninformation, the information processing device 100 detects predeterminedactions with a detection reference unique to the user, so as shown inFIG. 12, even in the event that a periodic action is performed within aset action region MA by an object which is not the user, the informationprocessing device 100 can prevent erroneous detection or excessivedetection of predetermined actions of users better than a case of usingreference detection information.

Referring to FIG. 9 again, an example of processing relating to theoperability improvement approaches at the information processing device100 according to an embodiment of the present disclosure will bedescribed. Upon the processing of step S108 being performed, theinformation processing device 100 determines whether or not apredetermined action has been detected in the set action regions MA(S110). In the event that a predetermined action is not detected in stepS110 in a set action region MA, the information processing device 100repeats the processing from step S102.

Also, in the event that determination is made in step S110 that apredetermined action has been detected in a set action region MA theinformation processing device 100 performs processing corresponding tothe detected action (S112). Here, the processing of step S112corresponds to the processing of (4) above (execution processing).

By performing the processing shown in FIG. 9 for example, theinformation processing device 100 can realize the processing of (1)(face region detection processing) through the processing of (4)(Execution processing) described above, relating to the firstoperability improvement approach. Accordingly, the informationprocessing device 100 can improve user operability by performing theprocessing shown in FIG. 9, for example.

1-2-2. Specific Example of Second Processing Relating to OperabilityImprovement Approach

FIG. 3 is a flowchart illustrating another example of processingrelating to the operability improvement approach at an informationprocessing device 100 according to an embodiment of the presentdisclosure, and illustrates an example of processing corresponding tothe second processing relating to the operability improvement approachdescribed above. In the following, description will be made withreference to an example of processing in a case of the informationprocessing device 100 determining a predetermined action made by a user,based on periodic components in an action region MA set as to the imagedimage.

In the same way as with step S100 in FIG. 9, the information processingdevice 100 determines whether or not to perform action detection of theuser (S300). In the event that determination is not made to performaction detection of the user in step S300, the information processingdevice 100 ends the processing relating to the operability improvementapproach. Note that the processing relating to the operabilityimprovement approach in FIG. 13 is not processing of a type which is notperformed again once it is ended, rather, the information processingdevice 100 can periodically/non-periodically repeat the processingrelating to the operability improvement approach shown in FIG. 13 eventhough the processing ends for the time being.

Also, in the event that determination is made to perform actiondetection of the user in step S300, in the same way as with step S102 inFIG. 9, the information processing device 100 performs face detectionbased on the input imaged image, and identifies a face region FA (S302).Now, the processing in step S302 corresponds to the processing of (I)above (face region detection processing). Also, in the event thatmultiple face regions FA are identified from the imaged image, such asin a case that there are multiple users included in the imaged image,the information processing device 100 performs the processing of thelater-described steps S304 through S316 for each of the identifiedmultiple face regions FA.

Upon performing the processing of step S302, in the same way as withstep S204 in FIG. 9, the information processing device 100 determineswhether or not a face region FA has been identified (S304). In the eventthat determination is not made that a face region FA has been identifiedin step S304, the information processing device 100 repeats theprocessing from step S300.

Also, in the event that determination is made that a face region FA hasbeen identified in step S304, in the same way as with step S106 in FIG.9, the information processing device 100 sets action regions MA based onthe face region FA (S306). Now, the processing in step S306 correspondsto the processing of (II) above (action region setting processing).

Upon action regions MA being set (or re-set) in step S306, in the sameway as with step S108 in FIG. 9, the information processing device 100detects predetermined actions in the set action regions MA (S308). Theinformation processing device 100 then determines whether or not apredetermined action has been detected within the set action regions MA(S310). In the event that determination is not made that a predeterminedaction has been detected in the set action regions MA in step S310, theinformation processing device 100 repeats the processing from step S302.

Also, in the event that determination is made that a predeterminedaction has been detected in the set action regions MA in step S310, theinformation processing device 100 notifies the user that a predeterminedaction has been detected (S412). Here, the processing of step S312corresponds to the processing of (IV) above (notification processing).

More specifically, as shown in FIGS. 4 through 8 for example, theinformation processing device 100 visually notifies the user thatdetermination has been made that a predetermined action has beenperformed by the user in an action region MA (or cause an externaldevice to make notification). Note that the information processingdevice 100 may also audibly notify the user (or cause an external deviceto make notification) by outputting audio indicating that thatdetermination has been made that a predetermined action has beenperformed by the user, for example, from an audio output device of itsown (e.g., a later-described audio output unit) or external audio outputdevice.

Also, in the event that determination is made that a predeterminedaction has been detected within a set action region MA in step S310, theinformation processing device 100 determines whether or not thepredetermined action has been continuously performed for a predeterminedperiod from determination that the predetermined action has beenperformed (S314). Now, the processing of steps S308, S310, and S314, areequivalent to the processing of (III) above (Action determinationprocessing).

Note that while FIG. 13 shows an example where the processing of stepS314 is performed after the processing of step S312, the informationprocessing device 100 can perform the processing of step S314 and theprocessing of step S312 independently, with the predetermined actionhaving been detected in a set action region MA in step S310 as atrigger.

In the event that determination is not made in step S414 that thepredetermined action has been continuously performed for a predeterminedperiod, the information processing device 100 repeats the processingfrom step S300. Now, an example of a case in which determination is notmade that the predetermined action has been continuously performed for apredetermined period is a case where the user has stopped his/her ownactions due to the notification from the processing in step S312.

Also, in the event that determination is made in step S314 that thepredetermined action has been continuously performed for a predeterminedperiod, the information processing device 100 performs processingcorresponding to the detected action (S316) in the same way as with stepS112 in FIG. 9. Here, the processing of step S316 corresponds to theprocessing of (V) above (execution processing).

By performing the processing shown in FIG. 13 for example, theinformation processing device 100 can realize the processing of (I)(face region detection processing) through the processing of (V)(execution processing) described above, relating to the operabilityimprovement approach.

Accordingly, the information processing device 100 can improve useroperability by performing the processing shown in FIG. 13, for example.

For example, by performing the processing shown in FIGS. 9 and 13, theinformation processing device 100 realizes the operability improvementapproach according to an embodiment of the present disclosure describedabove. It goes without saying that the processing relating to theoperability improvement approach with the information processing device100 according to an embodiment of the present disclosure is notrestricted to the processing shown in FIGS. 9 and 13.

2. Information Processing Device According to Embodiment of PresentDisclosure

Next, an example will be described of the configuration of aninformation processing device 100 according to an embodiment of thepresent disclosure, that is capable of performing processing relating tothe operability improvement approaches according to an embodiment of thepresent disclosure described above.

The configuration of the information processing device 100 according toan embodiment of the present disclosure will be described below withreference to an example of an information processing device 100 capableof executing the second processing relating to the operabilityimprovement approach described above. Note that the informationprocessing device 100 according to an embodiment of the presentdisclosure can realize the first processing according to the operabilityimprovement approach described above by having a configuration without alater-described notification processing unit, or a configurationincluding a notification processing unit but the notification processingnot functioning, for example. Also, the information processing device100 according to an embodiment of the present disclosure can selectivelyperform the first processing according to the operability improvementapproach and the second processing according to the operabilityimprovement approach as well, by having a configuration where thelater-described communication processing unit is made to selectivelyfunction based on user operation or the like.

First Embodiment

FIG. 14 is a block diagram illustrating an example of the configurationof an information processing device 100 according to a first embodimentof the present disclosure. Now, FIG. 14 shows as well a display system200 which performs display of an image (still image or moving image) ona display screen, and generating of imaged images by imaging in thedisplay direction of the display screen. Also, while FIG. 14 shows anexample where the display system 200 is made up of a display device 300which performs display and an imaging apparatus 400 which performsimaging, the configuration of the display system 200 according to anembodiment of the present disclosure is not restricted to theabove-described. For example, the display system 200 may be a devicewherein the display device 300 and the imaging apparatus 400 have beenintegrated (e.g., a display device having imaging functions).

Now, an imaged image which has imaged the user viewing the displayscreen is obtained by the display system 200 generating an imaged imagetaking images in the direction of display of the display screen.Accordingly, by the information processing device 100 processing theimaged image generated by the display system 200, the informationprocessing device 100 can easily perform processing such as identifyingface regions FA and setting action regions MA, detection ofpredetermined actions of the user, extraction of facial expressions, andso forth.

With reference to FIG. 14, the information processing device 100 has astorage unit 102, a communication unit 104, an input unit 106, adetecting unit 108, an action region setting unit 110, an actiondetermination unit 112, a notification processing unit 114, and aprocessing unit 116.

The information processing device 100 may further have a control unit(not shown), ROM (Read Only Memory (not shown)), RAM (Random AccessMemory (not shown)), and so forth, for example. The informationprocessing device 100 has the components interconnected by a bus servingas a data transfer path, for example. Here, the control unit (not shown)serves to control the entire information processing device 100. Notethat with the information processing device 100, the processing unit 116may serve as the control unit (not shown), for example. Also, ROM (notshown) stores programs which the control unit (not shown) uses, and datafor processing such as computation parameters and so forth, for example.The RAM (not shown) temporarily stores data and programs executed by thecontrol unit (not shown) for example.

Hardware Configuration Example of Information Processing Device 100

FIG. 15 is an explanatory diagram illustrating an example of thehardware configuration of an information processing device 100 accordingto an embodiment of the present disclosure. With reference to FIG. 15,the information processing device 100 includes, for example, an MPU 150,ROM 152, RAM 154, a recording medium 156, an input/output interface 158,an operation input device 160, a display device 162, and a communicationinterface 164. Also, the information processing device 100 has thecomponents interconnected with a bus 166 serving as a data transferpath, for example.

The MPU 150 is an integrated circuit where multiple circuits forrealizing various types of functions such as image processing and thelike are integrated, and so forth, and functions as a control unit (notshown) controlling the entire information processing device 100. Also,the MPU 150 serves as the detecting unit 108, action region setting unit110, an action determination unit 112, notification processing unit 114,and processing unit 116 in the information processing device 100. Also,the MPU 150 may serve as a later-described learning unit.

The ROM 152 control data such as stores programs and computationparameters and the like which the MPU 150 uses. The RAM 154 temporarilystores the programs and the like to be executed by the MPU 150, forexample.

The recording medium 156 functions as the storage unit 102, and storesvarious data such as reference detection information, individualdetection information, processing information, and so forth (ordatabases), applications, and so forth, for example. Examples of therecording medium 156 include magnetic storage media such as hard disksor the like, and nonvolatile memory such as EEPROM (ElectricallyErasable and Programmable Read Only Memory), flash memory, MRAM(Magnetoresistive Random Access Memory), FeRAM (Ferroelectric RandomAccess Memory), and PRAM (Phase change Random Access Memory). Also, theinformation processing device 100 may have a recording medium 156 whichis detachable from the information processing device 100.

Also, the input/output interface 158 connects the operation input device160 and display device 162, for example. Also, the input/outputinterface 158 can serve as the input unit 106. The operation inputdevice 160 serves as an operating unit (not shown), and the displaydevice 162 serves as a display unit (later-described). Now, examples ofthe input/output interface 158 include a USB (Universal Serial Bus)terminal, DVI (Digital Visual Interface) terminal, HDMI (High-DefinitionMultimedia Interface) terminal, various types of circuits, and so forth.Also, the operation input device 160 is provided on the informationprocessing device 100 for example, and is connected with theinput/output interface 158 inside the information processing device 100.Examples of the operation input device 160 include buttons, directionkeys, rotating selectors such as jog dials, combinations thereof, and soforth. Also, the display device 162 is provided on the informationprocessing device 100 for example, and is connected with theinput/output interface 158 inside the information processing device 100.Examples of the display device 162 include LCDs (Liquid Crystal Display)and organic ElectroLuminescence displays (also called OLED (OrganicLight Emitting Diode) displays). It goes without saying that theinput/output interface 158 can be connected external devices serving asexternal devices of the information processing device 100, such as anoperation input device (e.g., keyboard, mouse), display device (e.g.,external display device such as display device 300), imaging device(e.g., imaging apparatus 400 and so forth) and the like. Also, thedisplay device 162 may be a device capable of display and useroperations, such as a touch screen for example.

The communication interface 164 of the information processing device 100functions as the communication unit 104 for performing communication bywireless/cable with external devices such as the display device 300, orserver (not shown), and so forth for example, via the network (ordirectly). Examples of the communication interface 164 include acommunication antenna and RF circuit (wireless communication), IEEE802.15.1 port and transmission/reception circuit (wirelesscommunication), IEEE 802.11b port and transmission/reception circuit(wireless communication), or LAN terminal and transmission/receptioncircuit (cable communication), and so forth.

The information processing device 100 performs processing relating tothe operability improvement approaches according to an embodiment of thepresent disclosure, with the configuration shown in FIG. 15, forexample. Note that the hardware configuration of the informationprocessing device 100 according to an embodiment of the presentdisclosure is not restricted to the configuration shown in FIG. 15.

For example, the information processing device 100 may have an audiooutput device configured of a DSP (Digital Signal Processor) and anamplifier (amp) and speakers and the like, serving as an audio outputunit (later described). In the case of the above-described, theinformation processing device 100 can perform muting of audio and volumeadjustment for example, as processing corresponding to predeterminedactions of the user.

Also, the information processing device 100 may have an imaging deviceconfigured of a lens/imaging device and signal processing circuit,serving as an imaging unit (later described), for example. In the caseof the above-described, the information processing device 100 canprocess imaged images which it has generated itself. Now, thelens/imaging device is configured of an optical lens, and an imagesensor using multiple imaging devices such as CCD (Charge CoupledDevice) or CMOS (Complementary Metal Oxide Semiconductor) or the like,for example. The signal processing unit has, for example, an AGC(Automatic Gain Control) circuit or ADC (Analog to Digital Converter),converts analog signals generated by the imaging device into digitalsignals (image data), and performs various types of signal processing.Examples of signal processing which the signal processing circuitperforms include white balance correction processing, interpolationprocessing, color correction processing, gamma correction processing,YCbCr conversion processing, edge enhancement processing, codingprocessing, and so forth.

Also, the information processing device 100 may be of a configurationnot having the operation device 160 or display device 162 shown in FIG.15, for example.

Returning to FIG. 14, a first example of the configuration of theinformation processing device 100 according to an embodiment of thepresent disclosure will be described. The storage unit 102 is storagewhich the information processing device 100 has. Now, examples of thestorage unit 102 include magnetic recording media such as hard disks orthe like, and nonvolatile memory such as flash memory or the like.

Also, the storage unit 102 stores various data such as referencedetection information, individual detection information, processinginformation, and so forth (or databases), applications, and so forth,for example. FIG. 14 illustrates an example where reference detectioninformation 170, processing information 172, and recognition information174 are stored in the storage unit 102. Now, the recognition informationaccording to an embodiment of the present disclosure is informationrelating to a user, detected from an imaged image. Examples of therecognition information include the number of users included in theimaged image, expressions representing the feature of the user,information indicating the emotion of a user that has been estimated,presence or absence of body motions, and so forth, which is dataobtained by the information processing device 100 analyzing the imagedimage.

The communication unit 104 which the information processing device 100has performs communication by wireless/cable with external devices suchas the display device 300, or server (not shown), and so forth forexample, via the network (or directly). Also, the communication unit 104has communication thereof controlled by a control unit (not shown) andthe processing unit 116, for example. Examples of the communication unit104 here include a communication antenna and RF circuit LAN terminal andport and transmission/reception circuit, and so forth, but thecommunication unit 104 is not restricted to the above-described. Forexample, the communication unit 104 may have an optional configurationso as to be communicable with an external device via the network.

The input unit 106 receives the imaged image which the imaging apparatus400 has generated by imaging. The input unit 106 then hands the received(input) imaged image to the detecting unit 108. Now, examples of theinput unit 106 include a USB terminal, DVI terminal, HDMI terminal,various processing circuits, and so forth, but the input unit 106 is notrestricted to the above-described. For example, may have an optionalconfiguration so as to be capable of receiving imaged images from aneternal device by cable or wirelessly.

Note that the input unit 106 is not restricted to receiving imagedimages, and may receive data generated by a distance sensor or the likeprovided to the display device 300 (e.g., distance data indicating thedistance between the display screen and the user), for example. In theevent hat the above data is received, the input unit 106 hands the datato the processing unit 116. In the above-described case, the informationprocessing device 100 can perform processing corresponding topredetermined actions further based on distance data and the like handedat the later-described processing unit 116.

The detecting unit 108 has a leading role in the processing of (I) above(face region detection processing). More specifically, the detectingunit 108 detects the face region FA including the face of the user froman imaged image for example, and hands the information indicating thedetected (identified) face region FA and the imaged image to the actionregion setting unit 110.

Also, the detecting unit 108 may further detect a moving body in the setaction regions MA with the action region setting unit 110, and informthe action determination unit 112 whether or not a moving body has beendetected. In the above case, the action determination unit 112 canselectively perform later-described processing in the event that amoving body has been detected. Accordingly, with the informationprocessing device 100, reduction of the calculation load at theinformation processing device 100, and prevention of non-detection orerroneous detection predetermined actions of users, can be realized.

The action region setting unit 110 has a leading role in the processingof (II) above (action region setting processing). More specifically, inthe event that a face region FA is detected at the detecting unit 108for example, the action region setting unit 110 sets action regions MAnearby the face region FA, based on the face region FA. The actionregion setting unit 110 then hands the information of the set actionregions MA to the detecting unit 108 and the action determination unit112, for example. Also, the action region setting unit 110 hands theimaged image handed from the detecting unit 108, to the actiondetermination unit 112. Note that in the event that the configuration issuch that the input unit 106 and detecting unit 108 hand the imagedimage to the action determination unit 112, the action region settingunit 110 does not have to hand the imaged image to the actiondetermination unit 112.

Also, the action region setting unit 110 can extract face informationindicating the features of the face of the user from the face region FA,and re-set the action regions MA set based on the face region FA basedon face information (i.e., adjust the action regions MA).

The action determination unit 112 has a leading role in the processingof (III) above (action determination processing). More specifically, theaction determination unit 112 determines that a predetermined action hasbeen performed by the user in an action region MA which the actionregion setting unit 110 has set, based on the imaged image and referencedetection information 170. Now, the action region setting unit 110detects periodic components in the action regions MA and determines thata predetermined action has been performed based on the detection resultsof the periodic component in the action region MA and the periodiccomponent information in the reference detection information 170, forexample, but the processing at the action region setting unit 110 is notrestricted to the above-described.

In the event of determining that a predetermined action has beendetected within an action region MA, the action determination unit 112then hands the determination results to the notification processing unit114. Now, in the event of having determined that a predetermined actionhas been detected within an action region MA for example, the actiondetermination unit 112 selectively hands determination resultsindicating that the predetermined action has been detected to thenotification processing unit 114, but processing at the actiondetermination unit 112 is not restricted to the above-described. Forexample, the action determination unit 112 may hand determinationresults to the notification processing unit 114 each time processing isperformed relating to determination, regardless of the results of thedetermination. In the case of the above, the notification processingunit 114 will selectively perform later-described processing when thehanded determination results indicate that a predetermined action hasbeen detected within an action region MA.

Also, in the event that determination is made that a predeterminedaction has been detected within an action region MA set by the actionregion setting unit 110, the action determination unit 112 determineswhether the predetermined action has been performed continuously over apredetermined period from the determination that the predeterminedaction has been performed. Then, the action determination unit 112 handsdetermination results to the processing unit 116 indicating that apredetermined action has been performed continuously for a predeterminedperiod, in a case that determination is made that a predetermined actionhas been performed continuously for a predetermined period, for example.

Though not shown in FIG. 14, in the event that individual detectioninformation is stored in the storage unit 102, or in the event thatprocessing is performed collaboratively with an external device such asa server storing individual detection information, the actiondetermination unit 112 can selectively use the reference detectioninformation 170 or individual detection information to determine that apredetermined action has been performed by the user in an action regionMA.

Also, in the event of detecting periodic component in an action regionMA and determining that a predetermined action has been performed, theaction determination unit 112 does not have to detect periodic componentother than in the action region MA does not have to be detected. Due tothe above-described, the information processing device 100 can furtherreduce the load (calculation load) of processing relating to theoperability improvement approaches according to an embodiment of thepresent disclosure.

The notification processing unit 114 has a leading role in theprocessing of (IV) above (notification processing). More specifically,in the event that determination has been made at the actiondetermination unit 112 that a predetermined action has been performed,notification processing is performed to notify the user thatdetermination has been made that a predetermined action has beenperformed.

Now, the notification processing unit 114 transmits a processingexecution command to an external device via the communication unit 104,for notifying the external device that determination has been made thata predetermined action has been performed, as the notificationprocessing, for example. By the above, the information processing device100 can cause the display device 300 which is a display device, or thelike, to perform visual notification such as shown in FIGS. 4 through 8,or audible notification, for example.

The notification processing at the notification processing unit 114 isnot restricted to transmitting a processing execution command to anexternal device via the communication unit 104 for notifying theexternal device that determination has been made that a predeterminedaction has been performed. For example, in the event that theinformation processing device 100 has a display unit (described later),this display unit can be used to perform visual notification, and in theevent that the information processing device 100 has an audio outputunit (described later), this audio output unit can be used to performaudible notification. The above-described form will be described in alater-described third embodiment.

The processing unit 116 has a leading role in the processing of (v)above (cooperation determination processing). More specifically, in theevent that determination is made that predetermined actions have beenperformed continuously for a predetermined period at the actiondetermination unit 112, the processing unit 116, for example, performsprocessing corresponding to predetermined actions determined at theaction determination unit 112, based on the processing information 172.Now, predetermined actions and processing to be executed for example arerecorded in the processing information 172 in a correlated manner.Accordingly, by using the processing information 172 in which processingcorrelated with the meaning of the predetermined action has beenrecorded, the information processing device 100 can perform processingcorrelated to the meaning of the detected action (processing takingaffordance into consideration).

Also, the processing unit 116 can transmit a processing executioncommand to cause an external device to perform processing correspondingto the predetermined action via the communication unit 104, asprocessing corresponding to the predetermined action. Accordingly, theinformation processing device 100 can cause the display device 300 whichis an external device, or the like, to perform processing correspondingto the detected predetermined action.

Also, the processing unit 116 may perform processing such as recordingrecognition information in the storage unit 102, transmitting therecognition information 174 stored at the storage unit 102 to anexternal device such as a server (not shown) via the communication unit104, and so forth, for example.

With the configuration shown in FIG. 14, the information processingdevice 100 according to the first embodiment can realize the processingof (I) (face region detection processing) through the processing of (v)(execution processing) described above according to the secondoperability improvement approach. Accordingly, the informationprocessing device 100 can prevent deterioration of user convenience inthe communication system 1000 and improve user operability, by theconfiguration shown in FIG. 14 for example.

Modification of Information Processing Device 100 According to FirstEmbodiment

Note that the configuration of the information processing device 100according to the first embodiment of the present disclosure is notrestricted to the configuration shown in FIG. 14. For example, theinformation processing device 100 according to the first embodiment mayfurther have a device for realizing processing correlated with themeaning of a predetermined action performed by the user, such as anaudio output device or the like, a device relating to cooperativeprocessing with a communication object device 300, and so forth. Also,the information processing device 100 according to the first embodimentmay be arranged not having the storage unit 102 but rather obtaininginformation used for processing such as the reference detectioninformation 170 and processing information 172 from an external deviceas appropriate, so as to perform processing.

Second Embodiment

FIG. 16 is a block diagram illustrating an example of the configurationof an information processing device according to a second embodiment ofthe present disclosure. In FIG. 16, the display system 200 is showntogether, as with FIG. 14.

The information processing device 100 according to the second embodimentshown in FIG. 16 has basically the same configuration as the informationprocessing device 100 according to the first embodiment shown in FIG.14, but the information processing device 100 according to the secondembodiment further is provided with a learning unit 118 as compared tothe information processing device 100 according to the first embodimentshown in FIG. 14.

The learning unit 118 serves to manage individual detection information.For example, in the event of determination being made at the actiondetermination unit 112 that a predetermined action has been performed,the learning unit 118 updates the individual detection informationcorresponding to the determined predetermined action and the faceinformation extracted from the action region setting unit 110. Morespecifically, the learning unit 118 takes the information of theperiodic component detected at the action determination unit 112, forexample, as periodic component information in the individual detectioninformation corresponding to the determined predetermined action and theface information extracted by the action region setting unit 110.

Note that the learning unit 118 is not restricted to performing updatingof individual detection information in a case of determination beingmade at the action determination unit 112 that a predetermined actionhas been made, and for example, may perform updating of individualdetection information in the event that determination is made at theaction determination unit 112 that a predetermined action has beenperformed continuously for a predetermined period. Also, in the eventthat the detecting unit 108 has a function for detecting periodiccomponents in the action regions MA, the learning unit 118 may take theperiodic component information handed from the detecting unit 108 asperiodic component information in the individual detection information.

Now, in the event that determination is made at the action determinationunit 112 that a predetermined action has been performed (or in the eventthat determination is made that a predetermined action has beencontinuously performed for a predetermined period), the learning unit118 can automatically update the individual detection information, butthe processing at the learning unit 118 is not restricted to theabove-described. For example, the learning unit 118 can selectivelyupdate individual detection information based on operating signalscorresponding to user operations that are sent form an operating unit(not shown) or external operating signals transmitted from an externaloperating device such as a remote controller or the like.

Note that the management of individual detection information at thelearning unit 118 is not restricted to updating periodic componentinformation in the individual detection information described above. Forexample, the learning unit 118 can manage information unique to theuser, such as information relating to adjustment of region size of theaction regions MA and adjustment of position thereof, information of theperiod for determining the predetermined action of the user, and soforth, as individual detection information. Also, the learning unit 118can generate or update individual detection information based onoperating signals sent from an operating unit (not shown) in accordancewith user operations by way of a user interface for learning forexample, regardless of the determination results from the actiondetermination unit 112.

The information processing device 100 according to the second embodimenthas basically the same configuration as the information processingdevice 100 according to the first embodiment shown in FIG. 14.Accordingly, the information processing device 100 according to thesecond embodiment can realize the processing of (I) (face regiondetection processing) through the processing of (v) (executionprocessing) described above, according to the operability improvementapproach with the configuration illustrated in FIG. 16, in the same wayas with the information processing device 100 according to the firstembodiment. Accordingly, the information processing device 100 accordingto the second embodiment can improve user operability, with theconfiguration shown in FIG. 16, for example.

Also, by having the learning unit 118, the information processing device100 according to the second embodiment can manage information accordingto unique conditions of the user, such as for example, age, gender,physical features, and so forth, as individual detection information.Now, as described with reference to FIGS. 11 and 12, the informationprocessing device 100 prevents erroneous detection and excessivedetection of actions of a particular user corresponding to the faceinformation included in the individual detection information, bydetecting that a predetermined action has been performed usingindividual detection information. Accordingly, due to the learning unit118 managing individual detection information, the informationprocessing device 100 can improve the precision of determination in theevent of determining actions of a particular user corresponding toindividual detection information.

Modification of Information Processing Device 100 According to SecondEmbodiment

Note that the configuration of the information processing device 100according to the second embodiment of the present disclosure is notrestricted to the configuration shown in FIG. 16. For example, theinformation processing device 100 according to the second embodiment mayfurther have a device for realizing processing correlated with themeaning of a predetermined action performed by the user, as with thecase of the modification of the information processing device 100according to the first embodiment. Also, the information processingdevice 100 according to the second embodiment may obtain informationused for processing such as the reference detection information 170 andprocessing information 172 from an external device such as a server asappropriate, so as to perform processing, in the same way as with theinformation processing device 100 according to the first embodiment.

Third Embodiment

In the above description, a configuration has been illustrated as aninformation processing device 100 according to the first embodiment andan information processing device 100 according to the second embodiment,with the imaging apparatus 400 (e.g., external device) of the displaysystem 200 processing generated imaged images. However, theconfiguration of the information processing device 100 according to anembodiment of the present disclosure is not restricted to aconfiguration where an external device processes generated imagedimages.

FIG. 17 is a block diagram illustrating an example of the configurationof an information processing device 100 according to a third embodimentof the present disclosure.

The information processing device 100 according to the third embodimentof the present disclosure shown in FIG. 17 has basically the sameconfiguration as the information processing device 100 according to thefirst embodiment shown in FIG. 14, but the information processing device100 according to the third embodiment further is provided with animaging unit 120, a display unit 122, and an audio output unit 124 ascompared to the information processing device 100 according to the firstembodiment shown in FIG. 14.

The imaging unit 120 which the information processing device 100 hasgenerates imaged images by imaging in the display direction of thedisplay screen displayed on the display unit 122. The imaging unit 120sends imaged images to the detecting unit 108. By the imaging unit 120generating of imaged images by imaging in the display direction of thedisplay screen, an imaged image imaging the user watching the displayscreen is obtained. Accordingly, the information processing device 100can easily perform the processing of (I) (face region detectionprocessing) through (V) (execution processing) described above, such asidentifying of the face region FA, setting of action regions MA,detection of predetermined actions of the user, extracting facialexpressions, and so forth, for example.

Also, the imaging unit 120 is configured of one or two or more imagingdevices configured of a lens/imaging device and signal processingcircuit for example, but the configuration of the imaging unit 120 isnot restricted to the above-described.

The display unit 122 which the information processing device 100 hasdisplays various information and images on the display screen. Examplesof screens displayed on the display screen of the display unit 122include a content display screen for displaying contents, a screen fordisplaying imaged images generated by the imaging unit 120 (may besuperimposed with contents), an operating screen for allowing desiredactions to be performed as to the information processing device 100, andso forth.

The audio output unit 124 which the information processing device 100has outputs various audio such as audio indicated by contents, or systemaudio at the information processing device 100, for example. By havingthe audio output unit 124, the information processing device 100 canperform muting of audio and volume adjustment of audio output from theaudio output unit 124 for example, as processing corresponding topredetermined actions of the user which the processing unit 116performs.

Also, examples of the audio output unit 124 include an audio outputdevice configured of a DSP, and amplifier (amp) and speakers and thelike, but the configuration of the audio output unit 124 is notrestricted to the above-described.

The information processing device 100 according to the third embodimentof the present disclosure has basically the same configuration as theinformation processing device 100 according to the first embodimentshown in FIG. 14. Accordingly, the information processing device 100according to the third embodiment can realize the processing of (I)(face region detection processing) through the processing of (V)(execution processing) described above relating to the secondoperability improvement approach with the configuration shown in FIG.17, in the same way as the information processing device 100 accordingto the first embodiment. Accordingly, the information processing device100 according to the third embodiment can improve user operability, withthe configuration shown in FIG. 17, for example.

Modification of Information Processing Device 100 According to ThirdEmbodiment

The configuration of the information processing device 100 according tothe third embodiment of the present disclosure is not restricted to theconfiguration shown in FIG. 17. For example, the information processingdevice 100 according to the third embodiment may have the learning unit118 in the same way as with the information processing device 100according to the second embodiment. Also, the information processingdevice 100 according to the third embodiment may be of a configurationnot having the audio output unit 124 for example, or may further have adevice for realizing processing correlated to the meaning ofpredetermined actions of the user. Further, the information processingdevice 100 according to the third embodiment may be of a configurationwhich obtains information used for processing such as referencedetection information 170 and processing information 172 from anexternal device such as a server as appropriate, and performsprocessing, in the same way as with the modification of the informationprocessing device 100 according to the first embodiment.

As described above, the information processing device 100 according toan embodiment of the present disclosure performs, for example, theprocessing of (1) (face region detection processing) through theprocessing of (4) (execution processing) described above, or theprocessing of (I) (face region detection processing) through theprocessing of (V) (execution processing) described above, therebydetermining that a predetermined action has been performed by the user,and performing processing corresponding to the determined predeterminedaction. Now, the information processing device 100 sets action regionsMA nearby the face region FA identified based on the imaged image,detects (determines) predetermined actions of the user in the set actionregions MA, and performs processing corresponding to the detectedpredetermined actions. By setting the action regions MA in this way, theinformation processing device 100 can detect actions of the user byperforming image processing on a part of the regions in the imagedimage, rather than performing image processing on the entire imagedimage as with information processing device according to the relatedarts. Thus, the information processing device 100 can reduce thecalculation burden relating to detection of predetermined actions of theuser as compared to information processing device according to therelated arts, so with the information processing device 100, occurrenceof undesirable situations which can happen with information processingdevice according to the related arts such as described above, can beprevented. Accordingly, the information processing device 100 canprevent deterioration in user convenience can occur in the case of usingthe related art, and also can improve user operability.

Also, the information processing device 100 performs processingcorresponding to a predetermined action detected in an action region MAset nearby the face region FA, so erroneous detection of predeterminedactions performed by people or objects other than the user correspondingto the face region FA for example, can be reduced as compared toinformation processing device according to the related arts which takethe entire imaged image and the object of processing. Accordingly, theinformation processing device 100 can further improve user operabilityas compared to information processing device according to the relatedarts.

The information processing device 100 performs processing correlatedwith the meaning of the predetermined actions, as processingcorresponding to the predetermined actions detected at the actionregions MA set nearby the face region FA. As described above, due to theinformation processing device 100 performing processing correlated withthe meaning of the detected actions (processing taking affordance intoconsideration), the chance that the user will sense inconsistencybetween the user's own actions and the processing which the informationprocessing device 100 performs (or processing which the informationprocessing device 100 causes an external device to perform) can befurther reduced. Accordingly, the information processing device 100 canfurther improve user operability.

Further, in the event of determining that a predetermined operation hasbeen performed (in the event that a predetermined action has beendetected), the information processing device 100 can further performnotification processing in which the user is notified that determinationhas been made at the information processing device 100 that apredetermined action has been performed. Also, in the event ofperforming notification processing, in a case of determination havingbeen made that a predetermined action has been continuously performedfor a predetermined period from determination having been made that thepredetermined action has been performed, the information processingdevice 100 performs processing corresponding to the predetermined actionthat has been determined. Now, by the information processing device 100performing notification processing, the user can be made to comprehendthat the action which the user has performed has been recognized by theinformation processing device 100, so the user can cancel execution ofthe processing corresponding to this action at the informationprocessing device 100 by quitting his/her own action. Accordingly, inthe event the user desires to cancel execution of processingcorresponding to the action at the information processing device 100after the user has performed the predetermined action and theinformation processing device 100 has recognized this, or even in theevent that the information processing device 100 has happened toerroneously detect an action which the user has performed, for example,the user can control executing of processing at the informationprocessing device 100 corresponding to this action, by his/her ownaction.

Accordingly, by performing notification processing and furtherselectively performing processing corresponding to a predeterminedaction determined in the event that determination has been made that thepredetermined action has bee continuously performed for a predeterminedperiod, the information processing device 100 can prevent processingunintended by the user from being executed due to erroneous detection,and accordingly can further improve user convenience and operability.

While description has been made with the information processing device100 as an embodiment of the present disclosure, embodiments of thepresent disclosure are not restricted to this form. Embodiments of thepresent disclosure can be applied to various devices, such as forexample, computers such as PCs and servers, display devices such astelevision receivers and signage devices, portable communication devicessuch as cellular phones, video/audio players (or video/audiorecorder-players), game devices, and so forth.

Also, while description has been made with the display system 200 as anembodiment of the present disclosure, embodiments of the presentdisclosure are not restricted to this form. Embodiments of the presentdisclosure can be applied to various devices capable of displaying andimaging, such as for example, computers such as PCs, display devicessuch as television receivers and signage devices, portable communicationdevices such as cellular phones, and so forth. Note that the displaysystem 200 according to an embodiment of the present disclosure may be acombination of a device which performs display and a device whichperforms imaging. 3.

Program According Embodiment of Present Disclosure

User operability can be improved by a program for causing a computer tofunction as an information processing device according to an embodimentof the present disclosure (a program for realizing, for example, theprocessing of (1) (face region detection processing) through theprocessing of (4) (execution processing) described above, or a programfor realizing the processing of (I) (face region detection processing)through the processing of (V) (execution processing) described above).

While preferred embodiments of the present disclosure have beendescribed with reference to the attached drawings, it goes withoutsaying that the present disclosure is not restricted to these examples.It will be apparent to one skilled in the art that various alterationsand modifications can be made within the scope of the Claims, which areencompassed by the technical scope of the present disclosure as a matterof course.

For example, in FIGS. 14, 16, and 17, an example has been shown wherethe information processing device 100 according to the first throughthird embodiments of the present disclosure (including modifications)separately have the detecting unit 108, action region setting unit 110,action determination unit 112, notification processing unit 114, andprocessing unit 116, but the configuration of the information processingdevice according to an embodiment of the present disclosure is notrestricted to the above-described. For example, the informationprocessing device according to an embodiment of the present disclosuremay have the detecting unit 108, action region setting unit 110, actiondetermination unit 112, notification processing unit 114, and processingunit 116, formed as a single component (e.g., realized with a singleprocessing circuit). Also, the information processing device may haveoptional components integrated, such as for example, the detecting unit108 and the action region setting unit 110 formed as a single component(e.g., an image processing unit realized with a single image processingcircuit), and so forth. Further, the information processing device mayseparately have an information storage unit (not shown) for performingrecording of recognition information, for example.

Also, while in the above description, an illustration has been made of aprogram for causing a computer to function as an information processingdevice according to an embodiment of the present disclosure (computerprogram) being provided, an embodiment of the present disclosure mayfurther provide a storage medium in which the above-described programhas been stored, therewith.

The above-described configurations are an illustration of an example ofan embodiment of the present disclosure, and belong to the technicalscope of the present disclosure, as a matter of course.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A processing device, comprising: a storage forstoring detection information; and at least one processor operable to:detect a face region of an image; set at least one action region basedon position of the face region, wherein the at least one action regioncomprises at least one area outside the face region; extract faceinformation from the face region, wherein the face informationidentifies a user; determine individual detection information associatedwith the identified user from the stored detection information, whereinthe determined individual detection information comprises a plurality ofpredetermined actions associated with the identified user andcorresponding processing to be executed in response to the plurality ofpredetermined actions being performed by the identified user; compareimage data corresponding to the at least one action region with theindividual detection information to determine whether a predeterminedaction of the plurality of predetermined actions associated with theidentified user has been performed by the identified user; and executeprocessing corresponding to the determined predetermined action.
 2. Theprocessing device as recited in claim 1, wherein the at least oneprocessor is further operable to perform processing corresponding to thepredetermined action, when the predetermined action has been performedfor a predetermined period of time.
 3. The processing device as recitedin claim 1, wherein the at least one processor is further operable togenerate a notification when the predetermined action is performed,wherein the notification is a visual notification.
 4. The processingdevice as recited in claim 3, wherein the notification comprisesdisplaying the image on a display screen.
 5. The processing device asrecited in claim 3, wherein the notification comprises superimposing anotification object on the at least one action region.
 6. The processingdevice as recited in claim 3, wherein the notification comprises maskinga portion of the image.
 7. The processing device as recited in claim 3,wherein the notification comprises displaying an icon on another imagedisplayed on a display screen.
 8. The processing device as recited inclaim 3, wherein the notification comprises changing state of a lightingdevice provided on an outer portion of a display screen.
 9. Theprocessing device as recited in claim 3, wherein the notification is anaudio notification.
 10. The processing device as recited in claim 1,wherein the predetermined action corresponds to a periodic component ofthe image.
 11. The processing device as recited in claim 10, whereincomparing the image data corresponding to the at least one action regionwith the individual detection information comprises detecting theperiodic component in the at least one action region and detectingwhether the individual detection information correlates the periodiccomponent to the predetermined action.
 12. The processing device asrecited in claim 1, wherein the individual detection informationcomprises a plurality of periodic components of the image.
 13. Theprocessing device as recited in claim 1, wherein the at least oneprocessor is further operable to update the individual detectioninformation in response to detection of the predetermined action. 14.The processing device as recited in claim 1, wherein whether thepredetermined action has been performed is determined by detecting amoving body in the at least one action region.
 15. The processing deviceas recited in claim 1, wherein the at least one action region isrectangular, and the dimensions of the at least one action region aredetermined based on height and width of the face region.
 16. Theprocessing device as recited in claim 1, wherein setting the at leastone action region according to the position of the face region comprisesdetermining presence of the user corresponding to the extracted faceinformation, and setting at least one characteristic of the at least oneaction region based on the corresponding user.
 17. The processing deviceas recited in claim 1, wherein the at least one processor does notdetect the predetermined action that occurs outside the at least oneaction region.
 18. The processing device as recited in claim 1, whereinthe at least one action region is set based on the determined individualdetection information.
 19. The processing device as recited in claim 1,wherein when the individual detection information associated with theidentified user is not determined from the stored detection information,the image data is compared with reference detection information in thestored detection information.
 20. The processing device as recited inclaim 19, wherein the reference detection information is data serving asa reference of detection of the predetermined action.
 21. A processingmethod, comprising: detecting a face region of an image; setting atleast one action region based on position of the face region, whereinthe at least one action region comprises at least one area outside theface region; extracting face information from the face region, whereinthe face information identifies a user; determining individual detectioninformation associated with the identified user from the storeddetection information, wherein the determined individual detectioninformation comprises a plurality of predetermined actions associatedwith the identified user and corresponding processing to be executed inresponse to the plurality of predetermined actions being performed bythe identified user; comparing image data corresponding to the at leastone action region with the individual detection information to determinewhether a predetermined action of the plurality of predetermined actionsassociated with the identified user has been performed by the identifieduser; and execute processing corresponding to the determinedpredetermined action.
 22. A non-transitory computer-readable mediumstoring a computer-readable program for implementing a processingmethod, the processing method comprising: detecting a face region of animage; setting at least one action region based on position of the faceregion, wherein the at least one action region comprises at least onearea outside the face region; extracting face information from the faceregion, wherein the face information identifies a user; determiningindividual detection information associated with the identified userfrom detection information, wherein the determined individual detectioninformation comprises a plurality of predetermined actions associatedwith the identified user and corresponding processing to be executed inresponse to the plurality of predetermined actions being performed bythe identified user; comparing image data corresponding to the at leastone action region with the individual detection information to determinewhether a predetermined action of the plurality of predetermined actionsassociated with the identified user has been performed by the identifieduser; and execute processing corresponding to the determinedpredetermined action.